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1.0  SUMMARY 


The  goal  of  research  conducted  under  work  unit  WOPY  was  to  understand  insect  flight  for 
purposes  of  improving  agility,  autonomy,  robustness,  and  integrated  sensing  and 
processing  of  unmanned  aerial  vehicles.  This  goal  was  approached  using  a  comparative 
methodology  in  order  to  understand  general  principles  of  insect  flight  across  diverse 
species;  understand  environmental  variables  that  impact  natural  flight  of  insects; 
understand  how  insects  can  recover  from  flight  perturbations;  and  understand  the 
connection  between  flight,  sensor  capability,  neural  processing,  and  muscular  control. 

2.0  INTRODUCTION 

Insects  are  existence  proofs  for  agile,  robust,  autonomous  flight  that  minimizes  size, 
weight,  and  power  requirements,  aspects  that  are  desirable  for  human-engineered 
systems.  To  learn  design  principles  for  improved  sensors  and  guidance/control 
algorithms,  AFRL  studies  insect  sensors  and  flight.  The  current  research  effort  attempts 
to  connect  the  environmental  information  with  insect  flight  and  relate  that  to  insect  sensors 
and  processing.  The  goal  is  to  understand  insect  flight  for  purposes  of  improving  agility, 
autonomy,  robustness,  and  integrated  sensing  and  processing  of  unmanned  aerial 
vehicles. 

3.0  2014  ANNUAL  REPORT 

Indoor  laboratory  and  outdoor  laboratory  flights  of  insects  were  recorded  by  high-speed 
cameras  with  frame  rates  from  500-1000  Hz.  Indoor  laboratory  flights  were  recorded  in 
a  flight  chamber  measuring  2  m  x  1  m  x  1  m  and  lined  with  different  optic  flow  patterns. 
Outdoor  laboratory  flights  were  recorded  by  releasing  just  captured  insects  in  front  of 
high-speed  cameras  and  allowing  them  to  initiate  escape  flight.  The  goal  was  to  compare 
the  kinematics  of  each  flight  inside  the  laboratory  versus  outside  in  the  natural  world.  This 
effort  required  automating  the  tracking  of  the  insect  in  the  video  frames  because  the 
dataset  captured  is  extremely  large.  If  using  just  two  cameras  recording  at  1000  frames 
per  second,  the  study  cameras  are  capable  of  recording  8  seconds  of  data.  The  insect’s 
position  would  need  to  be  found  and  recorded  in  16000  frames  for  one  behavior  capture. 

David  Forester  (in-house  contractor)  started  work  automating  this  process  based  on  Ty 
Hedrick’s  algorithms  (Hedrick,  2008).  Figure  1  illustrates  challenges  in  adapting  these 
techniques  indoors.  Also  shown  in  Figure  1  are  early  behavior  recordings  of  Green  Darner 
dragonflies  (Anax  junius)  in  this  flight  chamber  by  Kaitlin  Fair.  Kaitlin’s  goal  was  to 
establish  repeatable  protocols  for  eliciting  flights  in  insects  large  enough  to  carry  a 
telemetry  recoding  chip  to  correlate  flight  kinematics,  responses  to  optic  flow  stimuli,  and 
muscle  potentials  (Harrison,  2011).  She  has  since  moved  on  to  studying  compressive 
sensing  for  her  PhD  but  hopes  to  return  to  working  on  this  starting  in  2017. 
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Figure  1 .  Early  tracking  results  illustrating  the  difficulty  in  tracking  a  small,  not  highly 
contrasted,  unmarked  flying  insect  against  a  background  with  changing  contrast  in  an 
automated  (or  semi-automated  way).  A)  Screen  capture  of  automated  tracking  program.  B) 
Trajectory  of  tracked  flying  insect  in  one  camera  view. 


Outdoor  flight  recordings  have  their  own  challenges,  as  illustrated  in  Figure  2.  It  is  also 
likely  that  the  objective  of  capturing  completely  natural  kinematics  is  not  being  reached, 
because  the  insects  are  manipulated  beforehand.  Future  efforts  will  move  towards 
completely  natural  conditions,  capturing  flight  from  insects  that  have  not  experienced  any 
interference  from  the  research  team. 
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Figure  2.  Outdoor  recording  screenshot  of  local  robberfly  (Diogmetes)  after  capture. 
Lighting  conditions  are  challenging  because  of  changing  weather  conditions.  The  sky  is  used 
as  a  background  in  an  attempt  to  increase  the  contrast  of  the  insect  and  background,  The 
robberfly  is  directly  above  the  release  point  in  this  screenshot. 


Gaze  stabilization  is  also  of  interest,  but,  as  can  be  seen  in  Figure  1,  the  head  of  the 
insect  is  not  easily  discernable  in  free  flight.  In  addition,  it  would  be  difficult  to  induce 
precise  behaviors  to  induce  a  gaze  stabilization  response.  Therefore,  efforts  were  started 
toward  characterizing  gaze  stabilization  in  insects  while  tethered.  The  stimulating  is  a 
rotating  horizon  line  produced  by  UV  and  green  LEDs  (Figure  2). 
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Figure  3.  Damselfly  gaze  stabilization  setup.  Panel  field  of  view  =  45  x  22.5  degrees.  Local 
species  of  damselflies,  robberflies,  and  dragonflies  have  all  been  tested  in  this  apparatus  and 

have  failed  to  respond  with  head  rotations. 
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4.0  2015  ANNUAL  REPORT 

4.1  Stabilization  of  the  head  in  flight 

Blowflies  stabilize  in  flight  by  moving  their  heads  and  bodies  (Hengstenberg  1993).  The 
visual  system  plays  a  role  in  this  multi-sensory  task.  Previous  work  has  shown  that  insects 
can  reduce  motion  blue  and  maintain  consistent  visual  perception  by  moving  their  head 
independently  of  their  body  by  employing  a  head  stabilization  reflex  which  is  a  control 
loop  that  rolls  the  head  with  respect  to  the  visual  environment  (Goulard  et  al  2015).  This 
has  been  studied  in  honeybees  (Boeddeker  2009),  wasps  (Violiet  201 3),  and  flies  (Krapp 
2012)  most  often  using  an  edge  as  visual  stimulation  to  a  tethered  insect. 

A  projector  dome  with  a  field  of  view  of  240  x  1 20  degrees  (Figure  2)  was  used  to  stimulate 
tethered  Ischnura  ramburii  damselflies  (Figure  1).  Early  recordings  used  stimulation  that 
was  readily  available — cityscapes  that  were  already  mapped  onto  the  dome  in  correct 
perspective  (Figure  3).  In  order  to  standardize  responses,  a  horizon  scene  was  built  that 
could  be  rotated  at  a  user-defined  speed,  transversing  a  user-defined  angle  (Figure  4). 


Figure  4.  Head  of  a  damselfly  showing  compound  eyes  and  ocelli  clearly.  B)  Tethered 

damselfly. 
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Figure  5.  Visible  wide  field  of  view  projector  for  stimulation  of  gaze  stabilization  response  in 

insects.  Field  of  view  =  240  x  120  degrees. 


Figure  6.  Initial  stimulation  for  damselflies  in  dome  with  field  of  view  =  240  x  120  degrees. 
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Figure  7. 


Designation  of  reference  frame  for  visual  stimulation. 


The  movement  of  the  damselflies  was  recorded  using  high-speed  cameras.  Work  is  now 
ongoing  to  quantify  the  head  rotation  and  body  positions  from  the  videos  (Figure  5). 


Figure  8.  Tethered  damselfly  in  the  dome,  screenshot  of  a  high-speed  recording.  Lines 

illustrate  desired  measurements. 
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In  2015,  we  manipulated  the  bounds  of  the  rotating  horizon  scene.  We  established  at 
what  angular  extent  a  center  disk  that  blacked  out  the  center  of  the  scene  stopped  the 
gaze  stabilization  response  in  a  damselfly  (Figure  6).  Then  we  established  the  angular 
bound  of  blacking  out  cone  in  stopping  a  stabilization  response  (Figure  7).  Combing  these 
measurements,  we  tested  a  ring  of  rotating  horizon  to  establish  whether  this  could  elicit 
a  stabilization  response  (Figure  8).  These  experimental  treatments  were  repeated  for  14 
damselflies  of  the  same  species  (mixed  gender)  and  the  results  were  found  to  be 
consistent  over  individuals. 


Figure  9.  Radius  of  blacked-out  center  is  enlarged  until  animal  stops  responding.  A)  Less 
than  140  degrees,  animal  still  responds  with  head  rotation.  B)  At  radius  of  140  degrees,  animal 

stops  responding,  [not  to  scale] 
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Figure  10.  Radius  of  blacked-out  “cone”  is  enlarged  until  animal  stops  responding.  A)  Greater 
than  155  degrees,  animal  still  responds  with  head  rotation.  B)  At  radius  of  155  degrees,  animal 

stops  responding. 


Figure  11.  Ring  of  stimulation  sufficient  and  necessary  for  head  movement  response  to 

rotating  horizon. 
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There  are  three  hypotheses  to  explain  the  consistent  response  in  Figure  8.  First,  the 
rotation  the  horizon  undergoes  in  the  center  is  different  than  at  the  edges  of  the  dome 
(Figure  9A).  At  the  edges  of  the  dome,  the  horizon  sweeps  a  larger  area  than  in  the  center. 
This  also  means  the  horizon  is  rotating  at  a  different  speed  at  the  edges  of  the  dome  than 
at  the  center.  The  damseiflies  could  have  a  particular  rotation  extent  or  frequency  that 
they  respond  to  for  stabilizing  their  gaze.  Rotations  above  and  below  this  frequency  or 
extent  would  not  elicit  this  response. 


A.  Rotation 


B.  Spatial  placement 


/Area  of  FOV  does  not  stimulate 
gaze  stabilization  circuit 

1  Area  of  FOV  stimulates 
gaze  stabilization  circuit 


C.  Intensity 


High  intensity 


Low  intensity 


Figure  12.  Hypotheses  to  explain  behavioral  response.  A)  Both  the  speed  of  the  horizon  and 
the  spatial  extent  of  the  horizon  movement  change  from  the  center  to  the  edges.  B)  The  field 
of  view  that  stimulates  the  gaze  stabilization  circuit  is  in  the  periphery  and  not  in  the  center.  C) 
The  intensity  is  decreased  when  areas  are  blacked  out,  and  the  animal  stops  responding  when 

intensity  it  too  low. 


The  second  hypothesis  to  explain  the  response  in  Figure  8  is  that  the  damselfly  optical 
processing  circuitry  uses  a  portion  of  the  field  of  view  of  the  eyes  for  gaze  stabilization 
(Figure  9B).  The  compartmentalization  of  this  response  might  correspond  to  differences 
in  the  resolution  of  the  compound  eye.  For  example,  the  upward  looking  part  of  the 
compound  eye  might  have  increased  resolution  to  look  and  track  prey  items  while  the 
periphery  would  be  used  to  track  large  scale  motion  of  the  visual  field  and  stabilize  the 
visual  sensors. 

The  third  hypothesis  to  explain  the  response  in  Figure  8  would  be  that  blacking  out 
portions  of  the  field  of  view  reduces  the  overall  intensity  of  stimulation  to  the  eyes  of  the 
damseiflies  and  therefore  decreases  the  response  accordingly  (Figure  9C).  The  damselfly 
optical  system  would  have  a  threshold  of  light  needed  to  utilize  their  gaze  stabilization 
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circuitry.  It  is  possible  that  blacking  out  portions  of  the  field  of  view  reduces  the  light  to 
levels  below  which  the  damselflies  cannot  respond  to  the  moving  horizon. 

In  2016,  these  three  hypotheses  (Figure  9)  were  tested  by  manipulating  the  horizon 
scene  in  ways  to  eliminate  these  confounding  explanations  (Figure  1 0)  for  rotation,  spatial 
placement,  and  intensity.  In  addition,  the  experiment  was  expanded  to  include  the 
following: 

•  Determine  behavioral  response  to  vertical  extent  of  stimulus. 

•  Determine  behavioral  response  threshold  to  texture  in  scene. 

•  Spectral  ERG  of  damselfly  eyes,  compound  and  ocellar. 

•  Alternatively  occlude  ocelli  and  compound  eyes  and  perform  behavior  tests  again. 

•  Test  other  insects  for  similar  response: 

o  Dragonflies 
o  Robberflies,  and 

o  Relatives  of  the  Ischnura  ramburii  damselfly. 

•  Measure  the  field  of  view  of  the  compound  eyes  and  ocelli. 


1.  Increase  the 
angular  extent  of 
rotation. 

2.  Increase  the  rate  of 
rotation. 


B.  Spatial  placement 


1 - >'■ 

Move  center  of  rotation 
towards  the  periphery. 


C.  Intensity 


Instead  of  black,  block  out  part 
of  the  scene  with  a  solid  color 
or  pattern. 


Figure  13.  Testing  hypotheses  to  explain  behavioral  response.  A)  Response  zone  will  shift  to 
match  a  particular  rotation  speed  or  spatial  displacement  of  the  horizon  line.  B)  The  field  of 
view  that  stimulates  the  gaze  stabilization  circuit  remains  stable  as  center  is  shifted.  C)  The 
response  zone  will  change  size  based  on  total  intensity  of  the  scene. 
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4.2  High-speed  Recordings  of  Natural  Flight  during  Very  Low  Light  Conditions 

High-speed  recordings  were  attempted  at  night  of  natural  flight  near  light  collection  setups 
(Figure  11).  LED  panels  emitting  light  in  the  infrared  were  used  on  the  ground  pointing 
skyward  next  to  a  light  sheet  reflecting  UV  light.  The  hope  was  that  the  UV  light  would 
attract  flying  insects.  The  insects  would  fly  through  the  infrared  light  from  the  LED  panels 
in  front  of  the  high-speed  cameras.  We  would  then  be  able  to  record  the  natural  flight  of 
these  insects  without  manipulating  them.  This  proved  to  be  challenging  in  execution  and 
no  usable  flights  were  captured. 


Figure  14.  High-speed  videography  setup  attempt  at  night  collecting  station. 


4.3  Challenges  to  Auto-Tracking  Insects 

Automation  saves  an  enormous  amount  of  man  hours  therefore  saving  money.  There  are 
multiple  challenges  to  tracking  insects  in  video  recordings: 

•  Dim  light 

•  Complex  backgrounds 

•  Small  subjects 

•  Unmarked  subjects 

•  Arbitrary  camera  angles,  insect  flies  directly  towards  camera  on  focal  axis 

A  summary  of  tracking  efforts  by  Eglin  so  far  include  the  use  of  Ty  Hedrick’s  code.  State 
of  the  art  automated  tracking  software  has  been  developed  in  Tyson  Hedrick’s  lab.  It  is 
the  field  standard,  and  it  is  easy  to  use  with  markers  and  large  specimens.  It  uses  one- 
point  tracking  per  specimen  but  can  track  multiple  specimens.  In-house  code  efforts  were 
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started  with  David  Forester  (in-house  contractor)  and  enabled  basic,  automated  tracking 
for  2D  of  insects  using  two-point  tracking.  This  was  better  than  Hedrick’s  code  of  one- 
point  tracking  because  it  enabled  us  to  obtain  body  axis  measurements.  David  Forester 
took  a  job  at  another  facility.  Kaitlin  Fair  (government  personnel)  then  expanded  the  user 
interface  and  contrast  conditions  for  2D  tracking  of  insects.  Kaitlin  is  now  working  on  her 
PhD  at  Georgia  Tech.  David  Richards  (in-house  contractor)  has  accomplished  the 
calibration  of  multiple  cameras  (based  on  Ty  Hedrick’s  code),  combined  multiple  2D  for 
3D  coordinates  of  insects.  He  has  also  compared  Trevor  Wardill’s  code  developed  to 
track  killer  flies  in  small,  brightly  lit  space  at  Oxford.  David  Richards  has  also  utilized  a 
simulation  to  compare  different  methods  for  accuracy;  this  simulation  is  called  IRMA. 
IRMA  is  a  simulation  tool  designed  in-house,  and  development  continues  via  contractors. 
It  slows  construction  of  imagery/movies  of  scenarios  for  different  sensors  using  a  visible 
static  camera  as  sensor.  The  scene  built  by  David  Richards  consists  of  a  floor  and 
tabletop  along  with  a  moving,  1  cm  cube  that  represents  the  flying  insect  (Figure  12). 


Left  camera  view  of  Irma  scene 

Frame  1000 


— 

CL) 

X 

'q. 


pixels 

Figure  15.  The  scene  consists  of  a  floor  and  tabletop  along  with  a  moving,  1  cm  cube  which 
represents  the  flying  insect.  The  cube’s  trajectory  consists  of  5,  200-frame  segments. 


Three  sources  of  error  have  been  identified  in  auto-tracked  trajectories  including 
quantization  error,  localization  error,  and  data  association  error  (Figure  13).  Ty  Hedrick 
tested  quantization  error  with  different  camera  positions  in  simulation  (Theriault,  2014). 
Tested  localization  error  by  adding  noise  in  simulation.  Tested  data  association  error  in 
hardware  testing  by  throwing  an  object  and  using  gravity  and  mass  to  check.  They  state 
their  error  as  being  less  than  body  length  of  a  bat  and  half  the  body  length  of  a  bird.  This 
would  be  several  body  lengths  of  an  insect.  For  insects,  especially  when  looking  at 
accelerations  where  errors  are  integrated  twice,  a  goal  would  be  to  reduce  the  error. 
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Figure  16.  Graphical  Calibration  Error  Results 


Using  IRMA  confirms  error  estimated  looking  at  different  camera  positions  and  adding 
noise.  The  goal  would  be  to  explore  when  and  how  different  algorithms  fail,  and  examples 
would  be  multiple  moving  objects,  moving  object  and  moving  background,  and/or 
background  changing  contrasts.  Also,  we  would  like  to  explore  observation  distance, 
pixels  on  target,  and  measure  position  error.  We  question  whether  it  would  it  be  possible 
to  define  a  requirement  for  maximum  observation  distance  and  minimum  pixels  on  target 
to  obtain  an  acceptable  position  error  (Figure  12). 


Figure  17.  Projected  relationships  between  position  error  (measured)  and  observation 

distance  and  pixels  on  target  (user  defined). 
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5.0  2016  ANNUAL  REPORT 

5.1  Outdoor  Completely  Natural  Flight  Capture 


Figure  18.  Natural  outdoor  flight  recording  setup 


Multiple  completely  natural  high-speed  recordings  of  flying  insects  were  attempted 
(Figure  1).  We  chose  to  focus  on  damseiflies  to  complement  the  gaze  stabilization  work 
(Figure  2).  It  would  be  useful  to  tether  insects  at  a  natural  body  angle,  but  this  element  is 
not  yet  known.  A  screenshot  of  one  of  these  recordings  illustrates  the  challenges  faced 
by  auto-tracking  algorithms  (Figure  2).  The  damseiflies  do  not  always  present  a  high 
contrast  to  the  background.  The  background  is  also  being  moved  by  the  wind  and  is  not 
constant.  We  are  still  working  on  automating  the  tracking  of  insects  in  these  natural 
conditions  (Figure  3). 
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Figure  19.  Damselflies  in  natural  flight  outdoors  with  complex  and  changing  background  due 

to  wind.  Screenshot  of  high-speed  recording. 
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Figure  20.  Auto-tracking  example  of  outside  flight  recordings  of  damselflies.  Movement  over 
five  frames.  The  blue  is  where  the  insect  is  now;  red  is  where  it  was  five  frames  ago. 


5.2  Gaze  Stabilization  of  Tethered  Damselflies 

For  technical  reasons,  we  chose  to  test  the  intensity  hypothesis  to  explain  the  gaze 
stabilization  response  found  in  2015.  Teal  would  be  used,  instead  of  black,  to  block  out 
portions  of  the  field  of  view  of  tethered  damselflies  and  the  angular  position  of  these 
bounds  noted  when  the  damselflies  stopped  responding  with  stabilization  movements 
(Figure  4).  This  would  either  increase  the  intensity  or  leave  it  unchanged.  However,  the 
dome  material  was  changed  because  the  old  fabric  was  tearing  and  the  projection  system 
updated  and  recalibrated.  The  damselflies  did  not  respond  to  the  new  system.  We 
switched  to  the  old  fabric  but  were  unable  to  switch  to  the  old  projection  system.  We 
therefore  needed  to  try  to  repeat  2015  experiments  in  addition  to  the  teal  bounds.  We 
successfully  obtained  five  individuals  that  would  respond  to  the  horizon  with  stabilization 
movements,  but  the  results  were  not  the  same  as  last  year  (Table  1).  Instead  of  a  ring  of 
stimulus  resulting,  the  bounds  of  the  disk  and  cone  overlap.  Further  testing  is  required 
before  these  results  can  be  interpreted. 
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Table  1.  Degree  responses  of  individuals  tested  with  black  and  teal  bounds. 


Black  Bounds 

Teal  Bounds 

Specimen 

Cone 

Disk 

Cone 

Disk 

1 

130 

100 

110 

130 

2 

110 

90 

90 

120 

3 

110 

90 

110 

110 

4 

120 

90 

120 

110 

5 

120 

90 

130 

90 

Figure  21 .  Dome  visual  stimulation  for  damselflies.  Teal  is  used  to  block  out  the  rotating 

horizon  line. 


5.3  Electroretinography  (ERG)  of  Damselfly 

ERGs  are  field  potential  recordings  from  the  eyes.  In  the  damselflies,  we  recorded  from 
the  compound  eyes.  The  stimulation  is  a  xenon  light  lamp  producing  light  from  the  UV  to 
near  IR  wavelengths.  A  monochromator  and  neutral  density  filters  allow  the  same  number 
of  photons  to  be  selected  at  a  particular  wavelength  to  be  used  as  spot  stimulation  onto 
the  eye.  An  electrode  in  the  eye  is  used  to  record  the  voltage  response.  The  voltage 
response  is  then  compared  to  a  baseline,  and  the  on/off  responses  are  measured  (Figure 
5).  In  addition,  a  green  LED  is  used  between  every  wavelength  measurement  to  assess 
the  accommodation  of  fatigue  of  the  individual  under  study. 
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Figure  22.  Analysis  of  voltage  response  using  electroretinography  recording  of  damselfly 

compound  eye. 


An  ERG  of  the  damselfly  illustrates  that  it  is  sensitive  to  most  of  the  visible  spectrum  in 
addition  to  UV  (Figure  6  left-hand  panels).  The  green  LED  does  not  illicit  a  consistently 
decreasing  response  over  experimental  time  (Figure  6  right-hand  panels),  and  therefore 
we  can  conclude  that  the  spectral  response  is  not  confounded  with  accommodation  or 
fatigue  of  the  insect.  The  strength  of  the  response  in  red  was  what  we  were  hoping  to 
find.  The  KH ILS  dome  has  a  strong  red  component  to  the  horizon  scene  in  the  ground 
(Figure  7).  In  the  future,  it  would  be  useful  to  look  for  spectral  differences  as  a  function  of 
location  in  the  eye,  e.g.,  dorsal  vs.  ventral.  Future  testing  would  also  include  testing  linear 
polarization  sensitivity  as  a  function  of  location  in  the  eye. 


Response  to  Spectral  Light  Stimulation  Green  Light  LED  Response  Over  Experimental  Steps 


Experimental  Steps 


Figure  23.  ERG  recordings  from  the  damselfly  to  spectral  light  in  addition  to  a  green  LED. 
One  green  light  LED  recording  was  taken  after  each  spectral  measurement. 
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Figure  24.  KHILS  Projector  Spectral  Characteristics.  A)  Ocean  spectrometer  measurement  of 
the  projected  sky  in  KHILS  bio-dome.  B)  Ocean  spectrometer  measurement  of  the  projected 

ground  in  KHILS  bio-dome. 


5.4  Compound  Eye  Resolution  of  Damselfly 

Attempted  measurements  using  a  highly  automated  goniometer  instrument  called 
FACETS  (Douglass  2016)  showed  some  change  in  facet  size  over  the  compound  eye  of 
a  damselfly  (Figure  8).  In  the  future,  when  measurements  are  complete,  we  will  use  them 
to  make  a  3D  model  (lens  size  and  3D  position).  That  model  will  then:  1)  map  the 
compound  eye  to  KHILS  dome  and  know  where  each  facet  is  looking,  2)  project  what 
each  facet  actually  sees,  and  3)  allow  us  to  project  selectively  to  each  facet. 
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Figure  25.  Measure  ommatidial  lens  array  with  FACETS. 
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